Forward-Mode Automatic Differentiation in Julia

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics. This document is an extended abstract that has been accepted for presentation at the AD2016 7th International Conference on Algorithmic Differentiation.Comment: 4 page

Dynamic automatic differentiation of GPU broadcast kernels

We show how forward-mode automatic differentiation (AD) can be employed within larger reverse-mode computations to dynamically differentiate broadcast operations in a GPU-friendly manner. Our technique fully exploits the broadcast Jacobian's inherent sparsity structure, and unlike a pure reverse-mode approach, this "mixed-mode" approach does not require a backwards pass over the broadcasted operation's subgraph, obviating the need for several reverse-mode-specific programmability restrictions on user-authored broadcast operations. Most notably, this approach allows broadcast fusion in primal code despite the presence of data-dependent control flow. We discuss an experiment in which a Julia implementation of our technique outperformed pure reverse-mode TensorFlow and Julia implementations for differentiating through broadcast operations within an HM-LSTM cell update calculation

Forward-mode Automatic Differentiation in Julia

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics

Forward-mode Automatic Differentiation in Julia

We present ForwardDiff, a Julia package for forward-mode automatic differentiation (AD) featuring performance competitive with low-level languages like C++. Unlike recently developed AD tools in other popular high-level languages such as Python and MATLAB, ForwardDiff takes advantage of just-in-time (JIT) compilation to transparently recompile AD-unaware user code, enabling efficient support for higher-order differentiation and differentiation using custom number types (including complex numbers). For gradient and Jacobian calculations, ForwardDiff provides a variant of vector-forward mode that avoids expensive heap allocation and makes better use of memory bandwidth than traditional vector mode. In our numerical experiments, we demonstrate that for nontrivially large dimensions, ForwardDiff's gradient computations can be faster than a reverse-mode implementation from the Python-based autograd package. We also illustrate how ForwardDiff is used effectively within JuMP, a modeling language for optimization. According to our usage statistics, 41 unique repositories on GitHub depend on ForwardDiff, with users from diverse fields such as astronomy, optimization, finite element analysis, and statistics

JuliaLabs/Cassette.jl: v0.3.13

<h2>Cassette v0.3.13</h2> <p><a href="https://github.com/JuliaLabs/Cassette.jl/compare/v0.3.12...v0.3.13">Diff since v0.3.12</a></p> <p><strong>Merged pull requests:</strong></p> <ul> <li>remove show on the generated context type (#206) (@KristofferC)</li> </ul&gt

JuliaDiff/ChainRules.jl: v1.57.0

<h2>ChainRules v1.57.0</h2> <p><a href="https://github.com/JuliaDiff/ChainRules.jl/compare/v1.56.0...v1.57.0">Diff since v1.56.0</a></p> <p><strong>Merged pull requests:</strong></p> <ul> <li>CompatHelper: add new compat entry for Statistics at version 1, (keep existing compat) (#748) (@github-actions[bot])</li> <li>Add rule for with_logger (#749) (@oxinabox)</li> <li>Add version bounds for stdlibs (#750) (@oxinabox)</li> </ul&gt

JuliaDiff/ChainRules.jl: v1.58.0

<h2>ChainRules v1.58.0</h2> <p><a href="https://github.com/JuliaDiff/ChainRules.jl/compare/v1.57.0...v1.58.0">Diff since v1.57.0</a></p> <p><strong>Merged pull requests:</strong></p> <ul> <li>muladd rule for zero tangent (#752) (@oxinabox)</li> </ul> <p><strong>Closed issues:</strong></p> <ul> <li>Basic functions don't work for CUDA (#751)</li> </ul&gt